Enabling Network-Aware Applications

Many high performance distributed applications use only a small fraction of their available bandwidth. A common cause of this problem is not a flaw in the application design, but rather improperly tuned network settings. Proper tuning techniques, such as setting the correct TCP buffers and using parallel streams, are well known in the networking community , but outside the networking community they are infrequently applied. In this paper, we describe a service that makes the task of network tuning trivial for application developers and users. Widespread use of this service should virtually eliminate a common stumbling block for high performance distributed applications. Internet backbone speeds have increased considerably in the last few years due to projects like Internet II and NGI. At the same time, projects like NTON [25] and SuperNet [34] are providing a preview of the near future of wide area networks. Unfortunately, distributed applications often do not take full advantage of these new high-speed networks. This is largely due to the fact that the applications use the default parameters for TCP, which have been consciously designed to sacrifice optimal throughput in exchange for fair sharing of bandwidth on congested networks. In order to overcome this limitation, distributed applications running over high-speed wide-area networks need to become " network-aware " [32][36], which means that they need to adjust their networking parameters and resource demands to the current network conditions. There exists a large body of work showing that good performance can be achieved using the proper tuning techniques. The most important technique is the use of the optimal TCP buffer size, and techniques for determining the optimal value for the TCP buffer size are described in [35]. Another important technique is to use parallel sockets, as described in [31]. Using a combination of these techniques, applications should be able to utilize all the available network bandwidth, which is demonstrated in [4], [1], and [16]. However, determining the correct tuning parameters can be quite difficult, especially for users or developers who are not network experts. The optimal TCP buffer size and number of parallel streams are different for every network path, vary over time, and vary depending on the configuration of the end hosts. There are several tools that help determine these values, such as iperf [14], pchar [26], pipechar [27], netspec [23], and nettest [22], but none of these include a client API, and all require some level …